Software testing is a process used to improve quality of developed computer software by detecting defects. In its simplest form, software testing tests at least one specific function of a software product under test. Herein, a “function” is an executable portion (e.g., program module, sub-routine, procedure, etc.) of a software product that uses input parameters to produce an observable result. For the sake of brevity, a specific “function under test” is called FUT herein.
Typically, software testing involves devising a set of input values for a FUT that will cause the FUT to exercise all or some of its computer-executable instructions. A tester verifies that the results produced by the test of the FUT are in accord with predefined expectations. Many different conventional testing approaches exist, and they range from informal and manual ad hoc testing to formal, controlled, and automated testing.
A major tool in the tester's toolbox is a “test harness.” FIG. 1 shows an example of a test harness 100 in a conventional software-testing scenario. FIG. 1 also shows a typical computer 110 having a primary memory 120 (e.g., volatile or non-volatile memory). This computer may be a general- or special-purpose computer in this illustrated traditional scenario. Running in this memory is the software product being tested. One function of the software product—the function being the focus of this exemplary scenario—is represented by Function Under Test (FUT) 130. For the sake of clarity, FIG. 1 shows only one function under test (e.g., FUT 130); however, FUT 130 may represent one or more functions being tested (concurrently or serially).
The test harness 100 operates in the memory 120 with the FUT 130 and, as illustrated in FIG. 1, “saddles” or operates with some degree of control over the FUT. Rather than receiving input as it normally would (e.g., input from a user-interface or from another function), the FUT 130 receives input from the test harhess 100. Also, rather than providing output as it normally would (e.g., output to a user-interface or to another function), the FUT 130 sends the output to the test harness 100. Conventional test harness technology (like test harness 100) is known to those of ordinary skill in the art of software testing.
Conventionally, the test harness 100 includes the logic for acquiring the input values for testing, logic for interacting with the FUT 130, and logic for acquiring the results of the test. Typically, the test harness 100 also includes logic for analyzing the results to determine if the actual results match the expected results. Conventionally, the logic of a test harness is typically custom written and is inextricably intertwined with the test values, which are expected to be used to test the FUT.
As shown in FIG. 1, the illustrated combination of the test harness 100 and the FUT 130 represents a specific and exemplary scenario under which the FUT is tested. Each different specific scenario is called a “test-case.” A test-case is defined by the FUT being tested, its testing logic, and specific combinations and/or permutations of input parameters being tested.
An example of a conventional test-case is represented by the following definition:
TABLE 1[PriorityOne]public void TestCaseNonParameterized( ){string commandToExecute =“base-string” + “test-string”;bool expectedResult = true;bool ignoreCase = true;ExecuteTest(commandToExecute,   expectedResult,ignoreCase);}
An exemplary conventional test-case named “TestCaseNonParameterized” is shown in Table 1. The testing logic is represented by the label “ExecuteTest.”
The actual input-parameter values for each test-case are provided by one or more test vector(s). A typical test vector is a file or data structure. While a test-case defines the testing conditions, the test vector supplies the actual input-parameter values for testing the FUT under the defined testing conditions of the test-case.
To illustrate, consider this example that tests functions of a “numeric calculator” software program:                Test-case: Testing “addition” function with input being two integers        Test Vector: Vector 1: Add 2+3; result should be 5 Vector 2: Add −1+4; result should be 3        Test-case: Testing “multiplication” function with input being at least one negative integer        Test Vector: Vector 1: Multiply 0*−4, result should be 0 Vector 2: Multiply −2*−4, result should be 8 Vector 3: Multiply −1*−1, result should be 1        
Effective testing of a FUT typically covers one or more test-cases and each test-case uses one or more rich test vectors. The set of values of a rich test vector typically includes a broad spectrum of plausible combinations and permutations of values for input parameters.
In the example shown in Table 1, the test-vector source (e.g., a file) is identified as being at a location specified in the test-case definition. Specifically, that location is “test-string” in this example. So, the input-parameter values for the FUT of this test-case are provided at the specified location of “test-string.”
FIG. 1 shows two exemplary test vectors: X 140 and Y 150, respectively. The values of test vectors may be supplied via manual data-entry or automatic generation. Regardless, these test vectors X and Y are a list of values stored in a datasource (e.g., file, data structure, or database). For example, the values in test vector X 140 may include a list of positive integers for an exemplary “addition” function test case, and the values in test vector Y 150 may include a list of some positive and some negative integers for an exemplary “multiplication” function test case.
A crosshatch area 145 represents values shared between the exemplary text vectors X and Y. For example, the crosshatch area may represent a set of shared positive integers that applies to test-cases for both the exemplary “addition” and “multiplication” functions. This sharing of values between test vectors may be accomplished numerous ways. For example, sharing may be accomplished via literal copying of data between vectors or via a cross-reference of common values.
For a particular test-case of the FUT 130 where the test-case is associated with test vector X 140, the test harness 100 acquires the values to be tested from test vector X. To test the FUT 130, the test harness 100 provides these acquired values to the FUT and analyzes the FUT's results to determine if the actual results match the expected results.
In the setting of formal, controlled, and automated testing (in which a test harness is typically used), a tester typically custom writes a test-case for each different FUT, each different combination and/or permutation of input parameters, and each contemplated test vector. Conventionally, the custom-written testing logic of each test-case is tightly intertwined with the input parameters that characterize the test-case. Moreover, the contents of test vector(s) are traditionally narrowly tailored to their associated test-case(s). Consequently, test-cases are typically rigidly defined and unable to adapt to new or updated test vectors.
The scope of testing increases, not only because of issues related to rigidity and inflexibility, but, also, because the number of possible defects in the FUT and the number of configurations of the FUT increases dramatically relative to the increase in the size and complexity of the FUT. Therefore, an effective test vector for a FUT may be quite large. Not only may the test vector be large, but it is also likely to grow, change, and evolve during the testing lifecycle of the FUT.